Doesn't think create a cache dir under the working directory? We need to add that to .gitignore.

It doesn't, it just downloads the files to the working directory.

Why the extra .data?

Without the extra .data I get AttributeError: 'NDData' object has no attribute 'shape' with these images.

Well, that's weird. It is supposed to be glue Data object. What has changed while I was away?

p.s. NDData having no ndim might be a design oversight. Even CCDData has ndim. I should ask astropy. See astropy/astropy#13775

Why change the color? The point is to show a color that is different from the default (red).

I was worried that green over Viridis colormap wasn't visible enough, but didn't realize when I changed this that it doesn't apply to the 50 random markers before, only the my_sky markers. I'll experiment and see what a non-red color with good visibility is.

I decided green was fine and reverted it.

Please explain what this is trying to do.

Maybe tell people how big these files are. I am still waiting for the first file to download and I am not sure how long it is supposed to take. 👀

To answer my own question (but should still be added to the notebook narrative, I think):

• jw02731-o001_t017_nircam_clear-f090w_i2d.fits 3.3GB
• jw02731-o001_t017_nircam_f444w-f470n_i2d.fits 789MB

In contrast, the HST/ACS files are around 161MB each (more or less).

Thinking about this more, even though it will be less impressive, I wonder if we should stick to smaller subarrays in the example notebook... 💭

Maybe more of a question for @havok2063 -- Is there no way to have MAST cache this somewhere outside of the source checkout tree or at least let user provide the cache directory? Currently, this downloads the big files into the notebooks directory, which poses two different risks:

1. If I do a git clean -xdf (which I do from time to time), I will have to redownload these again when I next run the notebook.
2. If someone accidentally added these to version control (you can overwrite .gitignore ignore if you are not careful though not easy but also not impossible), we will have to do some crazy git-fu to squash it out of git history (especially if it is not caught early).

Yes, you can specify a location of the download path with local_path, and you can set a cache boolean flag. If you are downloading files from the production domain, i.e.mast.stsci.edu, then you do not need to specify the base_url keyword. See https://astroquery.readthedocs.io/en/latest/api/astroquery.mast.ObservationsClass.html#astroquery.mast.ObservationsClass.download_file.

Thanks! In that case, I propose we add a local_path=Path.home() / 'mastDownloads' input to the Observations.download_file() calls. This is Windows-friendly, I checked.

@havok2063 It seems that local_path expects a filename, not a directory, contrary to the documentation. Should I report this as an issue in astroquery?

 [Failed]

WARNING: Found cached file /Users/rosteen/MAST_Downloads with size 64 that is different from expected size 603515520 [astroquery.query]

---------------------------------------------------------------------------
IsADirectoryError                         Traceback (most recent call last)
Cell In [4], line 11
      9 for fn in files:
     10     uri = f"mast:JWST/product/{fn}"
---> 11     result = Observations.download_file(uri, local_path=Path.home() / 'MAST_Downloads/', cache=True)
     12     with warnings.catch_warnings():
     13         warnings.simplefilter('ignore')

File ~/opt/anaconda3/envs/viz_dev/lib/python3.10/site-packages/astroquery/mast/observations.py:568, in ObservationsClass.download_file(self, uri, local_path, base_url, cache, cloud_only)
    565             self._download_file(data_url, local_path,
    566                                 cache=cache, head_safe=True, continuation=False)
    567 else:
--> 568     self._download_file(data_url, local_path,
    569                         cache=cache, head_safe=True, continuation=False)
    571 # check if file exists also this is where would perform md5,
    572 # and also check the filesize if the database reliably reported file sizes
    573 if (not os.path.isfile(local_path)) and (status != "SKIPPED"):

File ~/opt/anaconda3/envs/viz_dev/lib/python3.10/site-packages/astroquery/query.py:437, in BaseQuery._download_file(self, url, local_filepath, timeout, auth, continuation, cache, method, head_safe, **kwargs)
    433     progress_stream = io.StringIO()
    435 with ProgressBarOrSpinner(length, f'Downloading URL {url} to {local_filepath} ...',
    436                           file=progress_stream) as pb:
--> 437     with open(local_filepath, open_mode) as f:
    438         for block in response.iter_content(blocksize):
    439             f.write(block)

IsADirectoryError: [Errno 21] Is a directory: '/Users/rosteen/MAST_Downloads'

whoops, I missed this. Yeah I would report this as a bug in astroquery.

Would be nice to have this actually center on some feature on the image rather than just area of blank sky.

We might have to take the markers stuff out or delete the subsets first before doing this. It is taking too long on my machine with these new images.

Reduced this to 10 markers but still too slow on my machine.

Should we even give so many in the list? Someone is going to try to uncomment everything and their machine is going to 🔥

For the record, I only see small performance issues when loading and viewing all of these files. And hopefully those will be improved soon. I think it's useful to have them for anyone who wants to try, but maybe a warning saying they're large files and performance may be affected depending on system specs?

I'm updating to use the Cartwheel images instead, which are ~600 MB instead of 3.2 GB.

Did we want to provide a further description of why this notebook is different than the other ImvizExample/why we provide both?

Just running through the notebook, I'm getting an error on Imviz saying these coords are "Out of Bounds". Anyone else?

Ahh good call, this will fail if you've blinked and have data B active because it's smaller. I'll add a note about that.

tl;dr -- Why not MAST download here?

It's because the full observation has a ton of objects in the catalog and is way too slow to load. This is a version of the data made by @camipacifici that cuts down the number of objects to something manageable.

Less impressive but works better for me. Otherwise, it is still laggy and gives me IOPub errors even with glue-viz/glue-jupyter#325 installed.